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Background 


NORTHROP  GRUMMAN 


•  Six  Sigma  has  proven  to  be  a  powerful  enabler  for 
process  improvement 

-  CM  Ml  adoption 

-  Process  improvement  for  measurable  ROI 

-  Statistical  analysis 

•  This  presentation  will  focus  on  practical  tools  and 
techniques  for  use  by  systems  engineers 


Agenda 

•  What  is  Six  Sigma? 

•  How  does  it  apply  to  systems  engineering? 

•  Strategies  and  lessons  learned 
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Many  Approaches  to  Solving  the  Problems 
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•  Which  weaknesses  are  causing  my  problems? 

•  Which  strengths  may  mitigate  my  problems? 

•  Which  improvement  investments  offer  the  best  return? 


People 


Approaches  to  Process  I  mprovement 
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Data-Driven  (e.g.,  Six  Sigma,  Lean) 


•  Clarify  what  your  customer  wants 
(Voice  of  Customer) 

-  Critical  to  Quality  (CTQs) 

•  Determine  what  your  processes 
can  do  (Voice  of  Process) 

-  Statistical  Process  Control 

•  I  dentify  and  prioritize 
improvement  opportunities 

-  Causal  analysis  of  data 

•  Determine  where  your 
customers/  competitors  are  going 
(Voice  of  Business) 

-  Design  for  Six  Sigma 


Model-Driven  (e.g.,  CMM,  CMMi) 


•  Determine  the  industry  best 
practice 

-  Benchmarking,  models 

•  Compare  your  current  practices  to 
the  model 

-  Appraisal,  education 

•  I  dentify  and  prioritize 
improvement  opportunities 

-  Implementation 

-  Institutionalization 

•  Look  for  ways  to  optimize  the 
processes 


What  is  Six  Sigma? 
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•  Six  Sigma  is  a  management  philosophy  based  on 
meeting  business  objectives  by  reducing  variation 

-  A  disciplined,  data-driven  methodology  for  decision  making 
and  process  improvement 

•  To  increase  process  performance,  you  have  to  decrease 
variation 


Too  early  Too  late  Too  early  Too  late 


Spread  of  variation 
too  wide  compared  to 
specifications 


Spread  of  variation 
narrow  compared  to 
specifications 


■  Greater  predictability 
in  the  process 

■  Less  waste  and 
rework,  which  lowers 
costs 

■  Products  and  services 
that  perform  better 
and  last  longer 

■  Happier  customers 
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A  Typical  Six  Sigma  Project 
in  Systems  Engineering 
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•  The  organization  notes  that  systems  integration  has  been 
problematic  on  past  projects  (budget/  schedule  overruns) 

•  A  Six  Sigma  team  is  formed  to  scope  the  problem,  collect  data  from 
past  projects,  and  determine  the  root  cause(s) 

•  The  team's  analysis  of  the  historical  data  indicates  that  poorly 
understood  interface  requirements  account  for  90%  of  the 
overruns 

•  Procedures  and  criteria  for  a  peer  review  of  the  interface 
requirements  are  written,  using  best  practices  from  past  projects 

•  A  pilot  project  uses  the  new  peer  review  procedures  and  criteria, 
and  collects  data  to  verify  that  they  solve  the  problem 

•  The  organization's  standard  SE  process  and  training  is  modified  to 
incorporate  the  procedures  and  criteria,  to  prevent  similar 
problems  on  future  projects 
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Applicability  to  Engineering 
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•  System  engineering  processes  are  fuzzy 

-  Systems  engineering  "parts"  are  produced  using  processes 
lacking  predictable  mechanizations  assumed  for 
manufacturing  of  physical  parts 

-  Simple  variation  in  human  cognitive  processes  can  prevent 
rigorous  application  of  the  Six  Sigma  methodology 

-  Process  variation  can  never  be  eliminated  or  may  not  even 
reduced  below  a  moderate  level 

•  Results  often  cannot  be  measured  in  clear  $  savings 
returned  to  organization 

-  Value  is  seen  in  reduced  risk,  increased  customer 
satisfaction, 

more  competitive  bids,  . . . 
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Barriers  and  Challenges 
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•  Capturing  the  first,  'low  hanging  fruit"  makes  Six  Sigma 
implementation  look  easy... 

-  Clearer  problems,  simpler  solutions,  bigger  payoffs 

-  Little  need  for  coordination 

...but  later  projects  are  tougher 

-  Keeping  projects  appraised  of  similar  efforts,  past  and  current 

-  Focusing  on  “the  pain",  not  the  assumed  solution 

•  Engineering  process  measurements  are  often  difficult  to  analyze 

-  Dirty  (or  no)  data,  human  recording  problems 

-  May  necessitate  Define- Measure- Analyze- Measure- Analyze-etc. 

•  Must  demonstrate  the  value  of  quantitative  data  to  managers 

-  Management  style  -  reactive  vs.  proactive  vs.  quantitative 

-  Less  value  in  a  chaotic  environment 

-  Must  engage  customers 
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Additional  Challenges 
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•  Difficulty  in  collecting  subjective,  reliable  data 

-  Humans  are  prone  to  errors  and  can  bias  data 

-  E.g.,  the  time  spent  in  privately  reviewing  a  document 

•  Dynamic  nature  of  an  on-going  project 

-  Changes  in  schedule,  budget,  personnel,  etc.  corrupt  data 

•  Analysis  requires  that  complex  SE  processes  be  broken 
down  into  small,  repeatable  tasks 

-  E.g.,  peer  review 

•  Repeatable  process  data  requires  the 

project/  organization  to  define  (and  follow)  a  detailed 
process 
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Strategies  &  Lessons  Learned 
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Valuable  Tools  for  Engineers 
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•  Six  Sigma  provides  a  comprehensive  set  of  tools  for: 

-  Soliciting  and  understanding  customer  needs  (requirements, 
delighters,  perceptions  of  quality) 

-  Defining  and  improving  processes  (inputs/outputs, 
customer/suppliers,  essential/nonessential  activities, 
capability,  stability/ predictability) 

-  Understanding  data  (trends,  relationships,  variation) 

•  These  tools  can  be  used  even  if  your  organization  is  not 
implementing  Six  Sigma 


A  General  Purpose  Problem-Solving 
Methodology:  DMAIC 
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Problem  or  goal  statement  (Y) 


l 


Define 

- > 

Measure 

- > 

Analyze 

- > 

Improve 

- > 

Control 

•  Refine  problem  &  goal 
statements. 

•  Define  project  scope  & 
boundaries. 


•  An  improvement  journey  to  achieve  goals  and  resolve 
problems  by  discovering  and  understanding  relationships 
between  process  inputs  and  outputs,  such  as 


Y  =  f(defect  profile,  yield) 

=  f(review  rate,  method,  complexity . ) 


Toolkit 
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Define 

Measure 

Analyze 

1  mprove 

Control 

Benchmark 

Contract/ Charter 

Kano  Model 

Voice  of  the 

Customer 

Voice  of  the 

Business 

Quality  Function 
Deployment 

GQI M  and 

1  ndicator 
Templates 

Data  Collection 

Methods 

Measurement 

System 

Evaluation 

Cause  &  Effect 
Diagrams/  Matrix 

Failure  Modes  & 
Effects  Analysis 

Statistical  Inference 

Reliability  Analysis 

Root  Cause 

Analysis,  including 

5  Whys 

Hypothesis  Test 

Design  of 
Experiments 

Modeling 

ANOVA 

Tolerancing 

Robust  Design 

Systems  Thinking 

Decision  &  Risk 
Analysis 

PSM  Perform 
Analysis  Model 

Statistical 

Controls: 

•  Control 

Charts 

•  Time  Series 

methods 

Non-Statistical 

Controls: 

•  Procedural 

adherence 

•  Performance 
Mgmt 

•  Preventive 

measures 

7  Basic  Tools  (Histogram,  Scatter  Plot,  Run  Chart,  Flowchart,  Brainstorming,  Pareto  Chart),  Control  Charts  (for 
diagnostic  purposes).  Baseline,  Process  Flow  Map,  Project  Management,  "Management  by 
Fact",  Sampling  Techniques,  Survey  Methods,  Defect  Metrics 

Design  for  Six  Sigma  (e.g.,  DMADV) 
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Define 


Define 

project 

scope 

Establish 

formal 

project 


'i 

Measure 

— > 

Analyze 

- > 

Design 

— -> 

Verify 

Identify 

customers 

Explore  data 

Develop 

detailed 

design 

Research 

VOC 

Design 

solution 

Refine 

predicted 

performance 

Benchmark 

Predict 

performance 

Quantify 

CTQs 

Develop 

pilot 

Evaluate 

pilot 

Scale-up 

design 

Document 


The  Voices  of  Six  Sigma 
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Six  Sigma  includes  powerful  techniq 
for  understanding  the  problem  you  are 
trying  to  solve 


-  Voice  of  Customer 


-  Voice  of  Process 


-  Voice  of  Business 


•  These  techniques  are  useful  in  non- Six  Sigma  settings 
for  understanding: 

-  Customer  requirements  and  needs 


-  Process  performance  and  capability 


-  Business  priorities  and  trends 


Voice  of  Customer  (VOC) 
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•  A  process  used  to  capture  the  requirements/  feedback 
from  the  customer  (internal  or  external) 

-  Proactive  and  continuous 

-  Stated  and  unstated  needs 

-  "Critical  to  Quality  (CTQ)"-  What  does  the  customer  think 
are  the  critical  attributes  of  quality? 

•  Approaches: 

-  Customer  specifications 

-  I  nterviews,  surveys,  focus  groups 

-  Prototypes 

-  Bug  reports,  complaint  logs,  etc. 

-  House  of  Quality 


Requirements  Development 
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•  VOC  approaches  provide  powerful  methods  for  eliciting, 
analyzing,  and  validating  requirements 

•  Can  overcome  common  problems  by: 

-  Identifying  ALL  the  customers 

-  Identifying  ALL  their  requirements 

-  Probing  beyond  the  stated  requirements  for  needs 

-  Understanding  the  requirements  from  the  customers' 
perspective 

-  Recognizing  and  resolving  conflicts  between  requirements  or 
between  requirement  providers 


Identifying  Needed  Data 
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What  are  the  process  outputs  and  performance  measures? 

What  are  the  inputs?  What  are  the  relationships  among  outputs  and  inputs? 


•  We  need  to  find  out  what  contributes  to 

performance: 

•  What  are  the  process  outputs  (y's)  that  drive 
performance? 

•  What  are  key  process  inputs  (x's)  that  drive 
outputs  and  overall  performance? 


» Techniques  to  address  these  questions 


-segmentation  /  stratification 
-input  and  output  analysis 
-  y  to  x  trees 

-cause  &  effect  diagrams 


Using  these  techniques 
yields  a  list  of  relevant, 
hypothesized,  process 
factors  to  measure  and 
evaluate. 


Controlled  and  Uncontrolled  Factors 
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•  Controlled  factors  are  within  the  project  team's  scope 
of  authority  and  are  accessed  during  the  course  of  the 
project. 

•  Studying  their  influence  may  inform: 

-  cause-and-effect  work  during  Analyze 

-  solution  work  during  I  mprove 

-  monitor  and  control  work  during  Control 

•  Uncontrolled  factors  are  factors  we  do  not  or  cannot 
control 

•  We  need  to  acknowledge  their  presence  and,  if 
necessary,  characterize  their  influence  on  Y 

•  A  robust  process  is  insensitive  to  the  influence  of 
uncontrollable  factors 


Individual  Value 


Exercise  - 

What  is  Quantitative  Management? 
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•  Suppose  your  project 
conducted  several  peer 
reviews  of  similar  code,  and 
analyzed  the  results 

-  Mean  =  7.8  defects/ KSLOC 

-  +3a  =  11.60  defects/KSLOC 

-  -3a  =  4.001  defects/KSLOC 


•  What  would  you 
expect  the  next  peer 
review  to  produce  in 
terms  of  defects/ 
KSLOC? 

•  What  would  you 
think  if  a  review 
resulted  in  10 
defects/  KSLOC? 

•  3  defects/  KSLOC? 
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Exercise  -  What  is  Required  for  Quantitative 
Management? 
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•  What  is  needed  to  develop  the 
statistical  characterization  of  a 
process? 


UCL=1 1 .60 


Mean=7.8 


LCL=4.001 


The  process  has  to  be 
stable  (predictable) 

-  Process  must  be 
consistently  performed 

-  Complex  processes  may 
need  to  be  stratified 
(separated  into  simpler 
processes) 

There  has  to  be  enough 
data  points  to 
statistically  characterize 
the  process 

-  Processes  must  occur 
frequently  within  a  similar 
context  (project  or 
organization) 
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What  I  s  a  Control  Chart? 
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A  time-ordered  plot  of 
process  data  points  with 
a  centerline  based  on  the 
average  and  control  limits 
that  bound  the  expected 
range  of  variation 

Control  charts  are  one  of 
the  most  useful 
quantitative  tools  for 
understanding  variation 


Defects  per  Line  of  Code 


There  are  Many  Types  of  Control  Charts 
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U  Chart  of  Defect  Detected  in  Requirements  Definition 


UCL=0. 09633 


U=0. 07825 


LCL=0. 06017 
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Individual  Value 


What  is  Special  Cause  and 
Variation? 


12 

11 

10 

9 

8 

7 

6 

5 

4 
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UCL=11.60 
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Common  Cause 
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UCL=13.36 


Mean=7.467 


LCL=1.578 


•  Common  Cause  Variation 

-  Routine  variation  that 
comes  from  within  the 
process 

-  Caused  by  the  natural 
variation  in  the  process 

-  Predictable  (stable)  within 
a  range 


•  Special  Cause  Variation 

-  Assignable  variation  that 
comes  from  outside  the 
process 

-  Caused  by  a  unexpected 
variation  in  the  process 

-  Unpredictable 


What  Are  the  Key  Features  of  a  Control 
Chart? 
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Lower 

Control 

Limit 
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Defects  per  Line  of  Code 


What  Is  a  Stable ( Predictable)  Process? 
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U  Chart  of  Defects  Detected  in  Requirements  Definition 


UCL=0. 09633 


U=0. 07825 


LCL=0. 06017 
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What  if  the  Process  I  sn't  Stable? 
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•  You  may  be  able  to  explain 
out  of  limit  points  by 
observing  that  they  are  due 
to  an  variation  in  tne 
process 

-  E.g.,  peer  review  held  on 
Friday  afternoon 

-  You  can  eliminate  the  points 
from  the  data,  if  they  are  not 
part  of  the  process  you  are 
trying  to  predict 


•  You  may  be  able  to  stratify 
the  data  by  an  attribute  of 
the  process  or  attribute  of 
the  corresponding  work 
product 

-  E.g.,  different  styles  of  peer 
reviews,  peer  reviews  of 
different  types  of  work 
products 
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